Systems, methods, and computer-readable media for detecting image degradation during surgical procedures

ABSTRACT

Methods, systems, and computer-readable media for detecting image degradation during a surgical procedure are provided. A method includes receiving images of a surgical instrument; obtaining baseline images of an edge of the surgical instrument; comparing a characteristic of the images of the surgical instrument to a characteristic of the baseline images of the edge of the surgical instrument, the images of the surgical instrument being received subsequent to obtaining the baseline images of the edge of the surgical instrument and being received while the surgical instrument is disposed at a surgical site in a patient; determining whether the images of the surgical instrument are degraded, based on the comparing of the characteristic of the images of the surgical instrument and the characteristic of the baseline images of the surgical instrument; and generating an image degradation notification, in response to a determination that the images of the surgical instrument are degraded.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a Continuation of U.S. patent applicationSer. No. 17/256,422, filed Dec. 28, 2020, which is a U.S. National StageApplication filed under 35 U.S.C. § 371(a) of International PatentApplication Serial No. PCT/US2019/038869, filed Jun. 25, 2019, whichclaims the benefit of and priority to U.S. Provisional PatentApplication Ser. No. 62/693,530, filed Jul. 3, 2018, the entiredisclosure of each of which is incorporated by reference herein.

BACKGROUND

Robotic surgical systems may be used in minimally invasive surgicalprocedures. During a robotic surgical procedure, a surgeon controls arobotic surgical arm with a user interface at a remote surgeon console.The user interface allows the surgeon to manipulate a surgicalinstrument coupled to the robotic arm and to control a camera to receiveimages of a surgical site within a patient.

The surgeon console may include a stereoscopic display, sometimesreferred to as a three-dimensional (3D) display. In this regard,sometimes in conjunction with a corresponding pair of stereoscopiceyeglasses worn by the surgeon, such displays facilitate depthperception from an image by presenting the image to the surgeon as apair of distinct images separately provided to the left and right eyes,respectively. The stereoscopic display may display images provided by astereoscopic endoscope. Stereoscopic endoscopes employ two signal paths,usually a left-eye view and a right-eye view, which are matched andinterdigitated to generate a stereoscopic image. As does typically occurduring surgical procedures, biological material or otherprocedure-related material may occlude one of the lenses of thestereoscopic endoscope, thereby degrading the images provided to thedisplay. In the case of a stereoscopic endoscope, this degradation haspotential side effects upon the surgeon through the now mismatchedstereoscopic image pairs which can cause perception issues that tax thesurgeon's visual and cognitive pathways without the surgeon's awareness.This can result in degraded performance of the surgeon in perceiving andresponding to the observed stereoscopic information. Thus, it is usefulto be able to detect these mismatch situations and inform the surgeon ofthe need to correct the situation.

SUMMARY

Disclosed according to embodiments of the present disclosure are methodsfor detecting image degradation during a surgical procedure. In anaspect of the present disclosure, an illustrative method includesreceiving images of a surgical instrument, obtaining baseline images ofan edge of the surgical instrument, comparing a characteristic of theimages of the surgical instrument to a characteristic of the baselineimages of the edge of the surgical instrument, the images of thesurgical instrument being received subsequent to the obtaining of thebaseline images of the edge of the surgical instrument and beingreceived while the surgical instrument is disposed at a surgical site ina patient, determining whether the images of the surgical instrument aredegraded, based on the comparing of the characteristic of the images ofthe surgical instrument and the characteristic of the baseline images ofthe edge of the surgical instrument, and generating an image degradationnotification, in response to a determination that the images of thesurgical instrument are degraded.

In a further aspect of the present disclosure, the images of thesurgical instrument are received by an image capture device.

In another aspect of the present disclosure, the image capture device isa stereoscopic endoscope including a left-eye lens and a right-eye lens.

In a further aspect of the present disclosure, the characteristic of thebaseline images of the edge of the surgical instrument is obtainedduring an initial image capture device calibration.

In a further aspect of the present disclosure, the method furtherincludes periodically receiving images of the edge of the surgicalinstrument at a predetermined interval.

In another aspect of the present disclosure, the determination that theimages of the surgical instrument are degraded is based at least in parton a difference between the characteristic of the received images of thesurgical instrument and the characteristic of the baseline images of theedge of the surgical instrument being greater than a threshold value.

In yet another aspect, the method further includes determining thecharacteristic of the images of the surgical instrument by computing amodulation transfer function derived from the received images of thesurgical instrument.

Disclosed according to embodiments of the present disclosure are systemsfor detecting image degradation during a surgical procedure. In anaspect of the present disclosure, an illustrative system includes asurgical instrument including at least one edge, an image capture deviceconfigured to capture images of the surgical instrument, the images ofthe surgical instrument including a characteristic, a display device, atleast one processor coupled to the image capture device and the displaydevice, and a memory coupled to the at least one processor and havingstored thereon a characteristic of baseline images of the edge of thesurgical instrument, and instructions which, when executed by the atleast one processor, cause the at least one processor to obtain thecharacteristic of the baseline images of the edge of the surgicalinstrument, receive the images of the surgical instrument, compare acharacteristic of the images of the surgical instrument to thecharacteristic of the baseline images of the edge of the surgicalinstrument, the images of the surgical instrument being receivedsubsequent to the obtaining of the characteristic of the baseline imagesof the edge of the surgical instrument and being received while thesurgical instrument is disposed at a surgical site in a patient,determine whether the images of the surgical instrument are degraded,based on the comparing of the characteristic of the images of thesurgical instrument and the characteristic of the baseline images of theedge of the surgical instrument, and generate an image degradationnotification, in response to a determination that the images of thesurgical instrument are degraded.

Disclosed according to embodiments of the present disclosure arenon-transitory computer-readable media storing instructions fordetecting image degradation during a surgical procedure. In an aspect ofthe present disclosure, an illustrative non-transitory computer-readablemedium stores instructions which, when executed by a processor, causethe processor to receive images of a surgical instrument, obtainbaseline images of an edge of the surgical instrument, compare acharacteristic of the images of the surgical instrument to acharacteristic of the baseline images of the edge of the surgicalinstrument, the images of the surgical instrument being receivedsubsequent to obtaining the baseline images of the edge of the surgicalinstrument and being received while the surgical instrument is disposedat a surgical site in a patient, determine whether the images of thesurgical instrument are degraded, based on the comparison of thecharacteristic of the images of the surgical instrument and thecharacteristic of the baseline images of the edge of the surgicalinstrument, and generate an image degradation notification, in responseto a determination that the images of the surgical instrument aredegraded.

Any of the above aspects and embodiments of the present disclosure maybe combined without departing from the scope of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

Objects and features of the presently disclosed systems, methods, andcomputer-readable media will become apparent to those of ordinary skillin the art when descriptions of various embodiments thereof are readwith reference to the accompanying drawings, wherein:

FIG. 1 is a schematic diagram of a robotic surgical system, inaccordance with embodiments of the present disclosure;

FIG. 2 is a simplified perspective view of an image capture device and asurgical instrument, in accordance with embodiments of the presentdisclosure;

FIG. 3 is an image of a surgical site received from an image capturedevice, in accordance with the present disclosure;

FIG. 4 is a test pattern formed by a difference in luminance between asurgical instrument and surrounding anatomical material, in accordancewith an embodiment of the present disclosure;

FIG. 5 shows images of a test patterns which may be disposed on asurgical instrument, in accordance with another embodiment of thepresent disclosure;

FIG. 6 is a modulation transfer function graph for the test pattern ofFIG. 5 , in accordance with the present disclosure; and

FIG. 7 is a flowchart illustrating an example method of detecting imagedegradation, in accordance with the present disclosure.

DETAILED DESCRIPTION

The present disclosure generally relates to dynamic detection of imagedegradation, and providing associated notifications, during a surgicalprocedure. In order to determine an amount of degradation occurringduring a surgical procedure, endoscopic calibration techniques may beused. Prior to its use, and/or at the start of a surgical procedure, anendoscopic imaging system may be calibrated. Calibration, prior to use,includes the process of determining and recording base parameters, atpeak or near-peak operating conditions for the imaging system by using acalibration target. Calibration prior to the use of the endoscopicsystem thus provides a baseline metric of the endoscopic system beforethe occurrence of degradation. During a surgical procedure, a similartechnique as that employed during calibration may be used to determinecurrent parameters of the endoscopic imaging system. By automaticallyand dynamically comparing the current parameters with those of the baseparameters, endoscopic image degradation can be determined.

To that end, the present disclosure relates to systems, methods, andcomputer-readable media for enabling dynamic detection of imagedegradation of images of a surgical site during a surgical procedure,and for generating and displaying image degradation notifications duringthe surgical procedure. In this regard, during calibration of the imagecapture device, one or more baseline parameters of the image capturedevice, based on calibration targets such as test patterns and/or edgesof tools within the image capture device's field of view, are determinedand recorded. During the surgical procedure, images of the surgical siteare captured by the image capture device and provided to a computingdevice, such as a control console, for processing, using a similartechnique as that employed during calibration, to determine one or morecurrent parameters of the image capture device. By dynamically comparingthe current parameter(s) with the baseline parameter(s) of the imagecapture device, a determination regarding image degradation can be made.

As used herein, the terms “clinician,” “surgeon,” “observer,” and/or“viewer” generally refer to a user of a stereoscopic display devicedescribed herein. Additionally, although the terms “first eye” and“second eye” are used herein to refer to a left eye and a right eye,respectively, of a user, this use is provided by way of example andshould not be construed as limiting. Throughout this description, theterm “proximal” refers to the portion of the device or component thereofthat is farthest away from the patient (and thus closest to theclinician and/or surgical robot) and the term “distal” refers to theportion of the device or component thereof that is closest to thepatient (and thus furthest away from the clinician and/or surgicalrobot). Further, as referred herein, the term “signal path” (whetherright-eye or left-eye) refers to an optical-electrical-optical signalpath whereby images are captured optically, converted to anelectrical/digital signal to be transmitted, and again converted back toan optical image when received by a computing or display device. Whilethe illustrative embodiments below describe a robotic surgical system,those skilled in the art will recognize that the systems, methods, andcomputer-readable media described herein may also be used in othersurgical procedures, for example minimally-invasive surgical procedures,where a patient image capture device is used to capture images of asurgical site. Thus, the present disclosure is not intended to belimited to the exemplary embodiments using a robotic surgical system, asdescribed hereinbelow.

With reference to FIG. 1 , a robotic surgical system 1 is illustrated,and generally includes a surgical robot 10, a controller 30, at leastone processor 32, at least one memory 35, and a user interface console40. Surgical robot 10 generally includes one or more robotic arms 12 anda base 18. Robotic arms 12 may be in the form of arms or linkages eachhaving an end 14 that supports a surgical instrument 250. Surgicalinstrument 250 may be any type of instrument usable with robotic arm 12,such as a grasper, a knife, scissors, staplers, and/or the like. One ormore of robotic arms 12 may include an imaging device 16 for imaging asurgical site “S,” as also shown in FIG. 3 .

Controller 30 includes, and/or is communicatively coupled to, the atleast one processor 32 and memory 35, and may be integrated with userinterface 40 or provided as a standalone device within the operatingtheater. As described in further detail below, processor 32 executesinstructions (not shown) stored in memory 35 to perform steps and/orprocedures of the various embodiments described herein. As will beappreciated, the implementation of processor 32 and memory 35 isprovided by way of example only and should not be construed as limiting.For instance, steps and/or procedures of any of the embodiments of thepresent disclosure may be implemented by hardware components, firmwarecomponents, software components, and/or any combination thereof.

User interface 40 communicates with base 18 through controller 30 andincludes a display device 44 which is configured to display stereoscopicimages of the surgical site “S.” The images are captured by an imagingdevice (also referred to as “image capture device”) 16 and/or capturedby imaging devices that are positioned about the surgical theater (e.g.,an imaging device positioned adjacent patient “P,” and/or an imagingdevice 56 positioned at a distal end of an imaging arm 52). Imagingdevices (e.g., imaging devices 16, 56) may capture optical images,infra-red images, ultrasound images, X-ray images, thermal images,and/or any other known real-time images of surgical site “S.” Imagingdevices 16, 56 transmit captured images to controller 30 for processing,such as by processor 32, and transmits the captured and/or processedimages to display device 44 for display. In one embodiment, one or bothof imaging devices 16, 56 are stereoscopic endoscopes capable ofcapturing images of surgical site “S” via a right-eye lens 210 and aleft-eye lens 220, as further described in the description of FIG. 2 .

In further embodiments, user interface 40 may include or be associatedwith a portable display device 45, which, similar to display device 44,is configured to permit the user to view the stereoscopic images in amanner that the user perceives a three-dimensional and/or depth effectfrom the stereoscopic images. Portable display device 45 may be goggles,glasses, or any other portable or semi-portable display device, whichmay be used to allow the user to view stereoscopic images.

User interface 40 further includes input handles attached to gimbals 70which allow a clinician to manipulate surgical robot 10 (e.g., moverobotic arms 12, ends 14 of robotic arms 12, and/or surgical instrument250). Each of gimbals 70 is in communication with controller 30 andprocessor 32 to transmit control signals thereto and to receive feedbacksignals therefrom. Additionally or alternatively, each of gimbals 70 mayinclude control interfaces or input devices (not shown) which allow thesurgeon to manipulate (e.g., clamp, grasp, fire, open, close, rotate,thrust, slice, etc.) surgical instrument 250 supported at ends 14 ofrobotic arms 12.

Each of gimbals 70 is moveable to move ends 14 of robotic arms 12 withinsurgical site “S.” The stereoscopic images displayed on display device44 are oriented such that movement of gimbals 70 moves ends 14 ofrobotic arms 12 as viewed on display device 44. It will be appreciatedthat the orientation of the stereoscopic images on display device 44 maybe mirrored or rotated relative to a view from above patient “P.” Inaddition, it will be appreciated that the size of the stereoscopicimages displayed on display device 44 may be scaled to be larger orsmaller than the actual structures of surgical site “S” permitting thesurgeon to have a better view of structures within surgical site “S.” Asgimbal 70 is moved, surgical instrument 250 are moved within surgicalsite “S.” Movement of surgical instrument 250 may also include movementof ends 14 of robotic arms 12 which support surgical instrument 250. Inaddition to gimbals 70, one or more additional input devices may beincluded as part of user interface 40, such as a handle including aclutch switch, a touchpad, joystick, keyboard, mouse, or other computeraccessory, and/or a foot switch, pedal, trackball, or other actuatabledevice configured to translate physical movement from the clinician intosignals sent to processor 32.

As noted briefly above, to provide the user with a view of surgical site“S” during a surgical procedure, one or more of imaging devices 16, 56may be a stereoscopic endoscope disposed about surgical site “S,” suchas adjacent to surgical instrument 250, and configured to capture imagesof surgical site “S” to be displayed as stereoscopic images on displaydevice 44.

Turning now to FIG. 2 , a simplified, perspective view of a distal endof image capture device 200, such as imaging devices 16, 56, and adistal end of surgical instrument 250 are provided, in accordance withan embodiment of the present disclosure. Image capture device 200captures images of surgical site “S” via right-eye lens 210 and left-eyelens 220 to provide two distinct view point images that are transmittedto processor 32 for processing, and to display device 44 for display.Image capture device 200 includes a body 202, which includes, at itsdistal end, a lens assembly including right-eye lens 210 and left-eyelens 220. Right-eye lens 210 and left-eye lens 220 are each associatedwith a respective right-eye signal path and a left-eye signal path toprovide the captured images of surgical site “S” to processor 32 anddisplay device 44.

Surgical instrument 250 is illustrated as a vessel sealing device, whichincludes a body 242 having a surface 252. Those skilled in the art willrecognize that this illustrative surgical instrument 250 is providedmerely as an example, and that any other surgical tool or device may besubstituted for the illustrated vessel sealing device without departingfrom the scope of the present disclosure. In some embodiments, one ormore test patterns (for example, test pattern 255 a) are included onsurgical instrument 250. The test pattern is an identifier, for examplea unique identifier, that can be used to distinguish surgical instrument250 from a background during image processing. For example, as depictedin FIG. 2 , test pattern 255 a is disposed on surface 252 of surgicalinstrument 250 and may be black and white alternating solid bars. Inother embodiments, the test pattern may be any pattern, shapes, colors,or design providing contrast to be used in the detection of degradationof lenses 210, 220 of image capture device 200 and/or of images capturedby image capture device 200.

In another embodiment, as shown in FIG. 5 , test pattern 255 b may bemade up of increasingly thinner alternating solid black and white bars.Using such a pattern permits image capture device 200 to distinguishbetween where a black bar ends and a white bar begins. For example, andas described in greater detail in the description of FIG. 5 , as thebars become thinner from one end of test pattern 255 b to the other, thedistinction between the bars becomes increasingly more difficult todetect. In still another embodiment, the test pattern is a passive testpattern, which is not detectable by the user during a surgicalprocedure. For example, the passive test pattern may be disposed onsurface 252 of surgical instrument 250 in a manner that reflects lightin an infrared or ultraviolet range.

In still another embodiment, test patterns 255 a, 255 b extend along anentire outer area of surface 252 of surgical instrument 250.Alternatively, it is contemplated that test patterns 255 a, 255 b may belocated at discrete locations on surface 252 of surgical instrument 250.In a further embodiment, it is contemplated that any of test patterns255 a, 255 b may include specific markers, such as specialized shapes,which enhance the ability of image capture device 200 to determine thattest patterns 255 a, 255 b are present within the received image ofsurgical site “S.” In embodiments, it is contemplated that test patterns255 a, 255 b correspond to a type of surgical instrument so that eachdifferent surgical instrument 250 or type of surgical instrument 250(for example, ablation device, dissection device, stapler, vesselsealing device, etc.) has a unique test pattern, which can be used to,where necessary, identify surgical instrument 250.

In still another embodiment, a pseudo test pattern is generated using acontrast between surgical instrument 250 and the background of an imageof surgical instrument 250. The pseudo test pattern may be a proxy orsubstitute for an actual test pattern, such as test pattern 255 a, 255b. In some embodiments, one or more geometric characteristics ofsurgical instrument 250 may be used as a pseudo test pattern and/or maybe used to define a region of an image that acts as a pseudo testpattern. In one illustrative embodiment, one or more edges of surgicaltool 250 are used to define a pseudo test pattern, e.g., the regions ofthe image on both sides of an edge of surgical tool 250 define a pseudotest pattern. For example, referring now to FIG. 3 , image capturedevice 200 (not shown), using lenses 210, 220 continually capturesimages of surgical site “S.” Surgical site “S” includes anatomicalmaterial 230, which may include tissue, bone, blood vessels, and/orother biological material, and surgical instrument 250, which includesedges 251, is positioned within surgical site “S.” Although shown as asingle image, each of right-eye lens 210 and left-eye lens 220 capturesa different image of surgical site “S,” which is displayed by displaydevice 44 as a stereoscopic image of surgical site “S.” As brieflyalluded to above, test patterns need not be disposed on surface 252 ofsurgical instrument 250. Rather, a pseudo test pattern 255 c may becreated by the differences in luminance between surgical instrument 250extending to edges 251 and surrounding anatomical matter 230 of surgicalsite “S.” In particular, the edges 251 of surgical tool 250 as capturedin images by image capture device 200, may be substituted for a testpattern 255 a, 255 b, and, as further described below, the edges 251 ofsurgical instrument 250, and particularly the contrast/differencebetween the luminance of surgical instrument 250 proximate edges 251 andthe luminance of the background, e.g. surgical site “S,” may be used asa pseudo test pattern 225 c instead of actual test patterns 225 a, 225b. While differences in contrast between any edges of surgicalinstrument 250 may be used to define a pseudo test pattern, in someembodiments slanted edges, e.g. straight edges that are atnon-orthogonal angles as viewed by image capture device 200, are used.For example, analysis software may be used to look for and identifyedges that are not substantially distant from, or off of, horizontal orvertical, e.g., between about 5° and 10° away.

FIG. 4 illustrates another view of surgical site “S,” showing surgicalinstrument 250 placed within surgical site “S” and the surroundinganatomical material 230. A pseudo test pattern 255 c is created from asection of the image of surgical site “S” including at least one edge251 of surgical instrument 250. For example, a section an image ofsurgical site “S” including surgical instrument 250 and surroundinganatomical material 230 may be used to generate pseudo test pattern 255c, which may be based on the differences in luminance between surgicalinstrument 250, extending to edges 251, and surrounding anatomicalmaterial 230. For example, pseudo test pattern 255 c may include adarker section contrasted with a brighter section on opposing side ofedge 251 of surgical tool 250, thereby forming two zones of differentand contrasted luminance created by surface 252 of surgical instrument250 leading up to edges 251 of surgical instrument 250, and surroundinganatomical material 230. As will be appreciated by those skilled in theart, the brighter section and the darker section may respectivelycorrespond to the surface 252 of surgical tool 250 and the surroundinganatomical material 230, and, depending on the type of surgicalinstrument 250 and/or image capture device 200, may be reversed in termsof which correspond to the brighter and darker sections.

Referring now to FIG. 5 , an illustrative comparison is provided of animage of a test pattern captured during calibration of image capturedevice 200, and an image of the test pattern captured during a surgicalprocedure. For purpose of example, the illustrative comparison describedbelow will refer to an image of test pattern 255 b as disposed onsurgical instrument 250 captured during calibration of image capturedevice 200, and an image of test pattern 255 b′ captured during thesurgical procedure. However, those skilled in the art will recognizethat other test patterns, such as described above, may also be usedand/or substituted for test pattern 255 b without departing from thescope of the present disclosure. The image of test pattern 255 b is animage captured during calibration of image capture device 200, while theimage of test pattern 255 b′ is the image received by image capturedevice 200 at some point during a surgical procedure. Due to occlusionof and/or anatomical material 230 coming into contact with lenses 210,220, the image of test pattern 255 b′ is slightly blurred, or degraded,as compared to the image of test pattern 255 b. As test patterns 255 b,255 b′ are viewed from left to right, each can be broken into groups ofincreasingly thinner width black bars and white bars. The contrastbetween black bars and white bars of test patterns 255 b, 255 b′ iseasily distinguished for all of area “X.” However, within area “Y,” itbecomes increasingly difficult to distinguish the locations of blackbars, white bars, and areas of transition of each.

In an embodiment, the image of test pattern 255 b′ is used to determinea value of a modulation transfer function (“MTF”). For example, the MTFis used to provide a measure of the transfer of contrast of test pattern255 b′ and how well lenses 210, 220 of image capture device 200reproduce (or transfer) the detail of test pattern 255 b in a capturedimage. By obtaining an image of test pattern 255 b or other testpatterns (for example, test pattern 255 a, pseudo test pattern 255 c,and/or the like) during calibration of image capture device 200,baseline characteristics of the image of test pattern 255 b or othertest patterns can be determined. During use of image capture device 200,the ability of lenses 210, 220 of image capture device 200 to continueto reproduce (or transfer) the detail of the test pattern is determinedby applying the MTF to the received image of the test pattern to yieldone or more determined characteristics of the image of the test pattern.Over time, the determined characteristic(s) of the image of the testpattern may change due to degradation and/or occlusion of lenses 210,220 of image capture device 200, for example, as shown in the image oftest pattern 255 b′. As described below with reference to FIG. 7 , thebaseline characteristics of the test pattern can be compared to thedetermined characteristic(s) of test pattern 255 b′ as image capturedevice 200 is used during a surgical procedure in order to determinewhen image degradation has occurred.

In order to calculate the MTF for both the baseline characteristics ofthe test pattern and determined characteristic(s) of the test pattern,processor 32 is used to differentiate the black bars and white bars oftest pattern 255 b, or in the case of pseudo test pattern 255 c, thedarker sections and the brighter sections of the captured imageproximate edges 251 of surgical instrument 250. Based on the differencesbetween the black bars and white bars, or darker sections and brightersections, a sinusoidal graph can be plotted. Turning now to FIG. 6 , agraph of the contrast between black bars and white bars of test pattern255 b′ of FIG. 5 is illustrated. The graph of FIG. 6 is used tocalculate the MTF of test patterns 255 b′. In some embodiments, such asdepicted in the graph of FIG. 6 , an amplitude of 100 is assigned toareas containing black bars and an amplitude of −100 is assigned forareas containing white bars. As test pattern 255 b transitions betweenblack bars and white bars the plot of the graph increases or decreasesbetween the amplitude of −100 and 100.

Additionally, as the widths of the black bars and white bars of testpattern 255 b′ decrease, the ability of processor 32 to continue todifferentiate between black bars and white bars may likewise decrease.As such, the peak amplitudes of the graph may no longer approach peakamplitudes of −100 and 100 such that, eventually, the width of the blackbars and white bars becomes so thin that processor 32 can no longerdistinguish the black bars from white bars of test pattern 225 b′ andthe peak amplitude settles at 0. In some embodiments, a value of 100%may be assigned where the peak amplitude is between −100 and 100 and avalue of 0% may be assigned where the peak amplitudes settles at 0. Forpeak amplitudes between −100 and 100, a corresponding percentage between100% and 0% can be assigned. The MTF is typically expressed as apercentage of the distinguishable contrast between black bars and whitebars based on the line widths per picture height (LW/PH). Thus, as theline widths (width of black bars and white bars) of test pattern 255 b′become increasingly thinner, the percentage representing the contrastbetween the black and white bars decreases.

By assigning a value to the groups of increasingly thinner widths ofblack bars and white bars in the form of the LW/PH, the MTF percentagemay correspond to the LW/PH. For example, if a portion of the graph ofFIG. 6 ranges between peak amplitudes of 100 and −100 for a LW/PH assmall as 100 mm, the MTF is 100% at 100 mm. If another portion of thegraph of FIG. 6 ranges between peak amplitudes of 50 and −50 for theLW/PH at 50 mm, the MTF is 50% at 50 mm. These values of the MTF aredetermined by processor 32 and stored in memory 35.

In further embodiments, the MTF percentages may be converted frompercentages ranging between 0% to 100% to corresponding values rangingbetween 0 and 1, wherein 0 corresponds to processor 32 being incapableof distinguishing black bars from white bars, and 1 corresponds toprocessor 32 being able to completely distinguish the black bars fromwhite bars. It is further contemplated that processor 32 is capable ofdetermining the MTF of test pattern 255 b′ as it is received by way ofeach of right-eye lens 210 and left-eye lens 220, independently. Thus,image degradation for images captured by way of each of right-eye lens210 and left-eye lens 220 may be detected.

While the above description of FIGS. 5 and 6 refer to test pattern 255 bas an example, similar determinations and calculations may be performedwhen pseudo test pattern 255 c is substituted for test pattern 255 b.For example, in embodiments, one or more slanted edges 251 of surgicalinstrument 250 may be used to calculate the MTF. In such embodiments,there may not be a test pattern 255 a, 255 b disposed on surgicalinstrument 250. Instead, the contrast between brighter sections anddarker sections, corresponding to either the surface 252 proximate edge251 of surgical instrument 250 or the surrounding anatomical material230, respectively. Further information regarding computing the MTF basedon a slanted edge 251 of surgical instrument 250 is described inSlanted-Edge MTF for Digital Camera and Scanner Analysis, by Peter D.Burns, Proc. IS&T 2000 PICS Conference, pg. 135-138 (2000), andSharpness: What is it and how is it measured?,http://www.imatest.com/docs/sharpness (last visited Sep. 11, 2017), theentire contents of each of which are incorporated herein by reference.

FIG. 7 is flowchart of a method 700 of detecting image degradation ofimages captured by image capture device 200 during a surgical procedure,in accordance with embodiments of the present disclosure. Method 700 maybe implemented, at least in part, by processor 32 executing instructionsstored in memory 35. Additionally, the particular sequence of stepsshown in method 700 of FIG. 7 is provided by way of example and notlimitation. Thus, the steps of method 700 may be executed in sequencesother than the sequence shown in FIG. 7 without departing from the scopeof the present disclosure. Further, some steps shown in method 700 ofFIG. 7 may be concurrently executed with respect to one another insteadof sequentially executed with respect to one another, and/or may berepeated or omitted without departing from the scope of the presentdisclosure.

Generally, prior to the execution of method 700, calibration of imagecapture device 200 will have been performed. For example, during afactory calibration process, image capture device 200 may receive, orthe memory 32 may have stored therein, left-eye and right-eye images ofone or more test patterns. The test patterns may be similar to testpatterns 255 a, 255 b disposed on surface 252 of surgical device 250, ormay be data related to the contrast between the edges of surgical device250 and a surrounding environment, thereby generating a pseudo testpattern 255 c. In another example, the calibration process may beperformed at the start of a surgical procedure. In an embodiment inwhich images of the test patterns (“baseline images”) are received byimage capture device 200 during calibration, a pattern analysisfunction, such as the modulation transfer function (MTF), is applied tothe test pattern to calculate output values. The output values mayrepresent the sharpness or clarity of a transition across featuresmaking up the test pattern and may be expressed as line widths perpicture height (LW/PH). The calculated output values may be included asone or more characteristics of the baseline images captured by theright-eye lens 210 and the left-eye lens 220 of the image capture device200 (“baseline characteristics”), and can be stored in the memory 32 forlater use, as will be described in detail below. Alternatively, thesebaseline characteristics may be known values that are stored in thememory 32. Additionally, during a calibration process, image capturedevice 200 may store the baseline characteristics of the test pattern inmemory.

In any case, surgical system 10 is configured to permit the user tobegin the surgical procedure within surgical site “S,” at step 705. Forexample, in the case of a robotic surgical procedure, the user moves thegimbals 70 to thereby position image capture device 200 and surgicalinstrument 250 about surgical site “S”. In some embodiments, the fieldof view of image capture device 200 may initially be aligned withsurgical instrument 250 to enable image capture device 200 to captureimages of surgical instrument 250, and particularly test patterns 255 a,255 b, via right-eye lens 210 and left-eye lens 220, respectively.Alternatively, as will be appreciated by those skilled in the art, innon-robotic minimally-invasive surgical procedures, image capture device200 and surgical instrument 250 may be positioned manually aboutsurgical site “S.”

Once suitably positioned, image capture device 200 captures images ofsurgical site “S,” and transmits the captured images to controller 30 atstep 710. In addition to tissue and surrounding anatomical material 230on which the surgical procedure is being performed, the captured imagesshow surgical instrument 250 as it is being manipulated by the user. Thecaptured images may be stereoscopic images, that is, left-eye images andright-eye images.

After receiving the captured images from image capture device 200,processor 32 processes the captured images to identify a test pattern,at step 715. As described herein, it is contemplated that the testpattern may be a pattern disposed on surface 252 of surgical instrument250 or a test pattern formed by the contrast been surgical instrument250 and surrounding anatomical material 230 about a slanted edge 251 ofsurgical instrument 250. As noted above, examples of test patternsinclude but are not limited to test patterns 255 a, 255 b and pseudotest pattern 255 c, and/or the like. In an embodiment, a suitablealgorithm is applied to the left-eye images and the right-eye images ofsurgical site “S” to output a result, which is analyzed by processor 32to detect a presence of the test pattern.

Optionally, in an embodiment where test pattern 255 b is disposed onsurgical instrument 250, at step 720, after identifying test pattern 255b′, a determination is made as to whether test pattern 255 b′ matches aknown test pattern. For example, a database of known test patterns maybe stored in memory 35, for example, in a look-up table. The images oftest pattern 255 b′ captured by image capture device 200 is comparedwith the known test pattern images stored in memory 35. In anembodiment, each of the known test patterns is associated with adifferent surgical instrument. As such, matching test pattern 255 b′with the known test pattern further includes identifying the surgicalinstrument corresponding to the known test pattern. In an embodiment,the identification of the surgical instrument is provided to the uservia display device 44 or through an audio device. For illustrativepurposes and provided by way of example, pseudo test pattern 255 c, thatis, the slanted edges 251 of surgical instrument 250, is used as theexemplary test pattern for the remaining description of FIG. 7 .

At step 725, the one or more baseline characteristics of the testpattern (“characteristic(s) of the baseline image of the test pattern”),generated and/or calculated during calibration, are obtained. Inembodiments where the calibration process is not performed prior to thestart of the surgical procedure, the calibration process may beperformed at step 725. In other embodiments, as noted above, thecharacteristic(s) of the baseline images of the test pattern may bestored in memory 35. In such embodiments, the correspondingcharacteristic(s) of the baseline image of the test pattern areretrieved from a lookup table and/or database stored in memory 35.

At step 730, the image of pseudo test pattern 255 c received at step 715is analyzed and one or more characteristics are determined from theimages of pseudo test pattern 255 c. In an embodiment, a MTF iscalculated for the images of pseudo test pattern 255 c in order todetermine the characteristics of the images of pseudo test pattern 255c. The determined characteristic(s) of the images of pseudo test pattern255 c may be in the form of a percentage at LW/PH. For example, at step730, the MTF may yield values of 100% at 105 mm and 45% at 50 mm. Inother embodiments, various other types of analysis functions may beapplied to the images of pseudo test pattern 255 c. In any case, step730 is continuously reiterated, so that changes in the determinedcharacteristic(s) of the images of pseudo test pattern 255 c may bedetected from the images of pseudo test pattern 255 c over time.

In step 735, the determined characteristic(s) of the images of pseudotest pattern 255 c are compared to the one or more characteristics ofthe baseline images of pseudo test pattern 255 c. In one embodiment, itis contemplated that either the percentage or the LW/PH are selected ata specific value for the characteristic(s) of the baseline images ofpseudo test pattern 255 c to be compared with percentages or LW/PH ofthe determined characteristic(s) of images of pseudo test pattern 255 c.For example, if the LW/PH is selected for the characteristic(s) of thebaseline images of pseudo test pattern 255 c at a specific value of 50mm and the corresponding percentage is 50% (thus yielding a MTF graphamplitude ranging from 50 to −50 at LW/PH of 50 mm), as described abovein the description of FIG. 6 , then at step 735 the determinedcharacteristic(s) of images of pseudo test pattern 255 c at LW/PH of 55m (45% at 50 mm, as described in step 30) are compared to 50% at 50 mm.Thus, using the examples above, the comparison between thecharacteristic(s) of the baseline images of pseudo test pattern 255 cand the determined characteristic(s) of pseudo test pattern 255 c yieldsa 5% difference.

In an embodiment, at step 740, a determination is made as to whether thedifference between the characteristic(s) of the baseline images ofpseudo test pattern 255 c and the determined characteristic(s) of theimages of pseudo test pattern 255 c is greater than a predeterminedthreshold. The predetermined threshold is a percentage decrease betweenthe characteristic(s) of the baseline images of pseudo test pattern 255c and the determined characteristic(s) of pseudo test pattern 255 c. Forexample, a predetermined threshold of the modulation function maytranslate to a decrease of a 15% of line widths per picture height(LW/PH), which may indicate image marrying from material on the lens ofone or both images. If the difference is not greater than apredetermined threshold (“NO” at step 740), method 700 proceeds to step745, where new images of surgical site “S” are received. Next, at step747, similar to step 715, the new images of surgical site “S” areprocessed to identify pseudo test pattern 255 c. Following step 747,method 700 returns to step 730 where the received images of pseudo testpattern 255 c are analyzed by processor 32 and one or morecharacteristics of the images of pseudo test pattern 255 c aredetermined.

It is contemplated that steps 730 through step 740 and returning to step730 may be performed iteratively and repeated at regular intervals. Inone embodiment, it is contemplated that processor 32 will proceed fromsteps 730 through step 740 returning to step 730 processing newlyreceived images of surgical site “S” at 60 Hz, possibly 30 Hz, and evenas low as 10 Hz. Thus, relatively frequesly, new images of surgical site“S” is received, processed, and a determination made as to whether thedifference between the characteristic(s) of the baseline image of pseudotest pattern 255 c and the determined characteristic(s) of the images ofpseudo test pattern 255 c, is greater than a predetermined threshold. Inother embodiments, based on the surgical procedure, the intervals fromsteps 730 through step 740 returning to step 730 may be shortened inorder to increase the frequency of the determination of imagedegradation, for example, every half, quarter, or one-tenth of a second.

In a further embodiment, a determination is made as to whether a trendis detected from the differences determined at step 735. It iscontemplated that the differences from step 735 are stored in memory 35,and the data is monitored to determine the trend. For example, processor32 may determine that image degradation has likely occurred due totissue occluding one or both of lenses 210, 220, where image degradationoccurs rapidly following a small number of passes from steps 730 through740 and returning to step 730. Alternatively, processor 32 may determinethat image degradation has occurred due to a gradual build-up of fluidor other anatomical material where image degradation occurs more slowly.

If, at step 740, it is determined that the difference between thecharacteristic(s) of the baseline images of pseudo test pattern 255 cand the determined characteristic(s) of the images of pseudo testpattern 255 c is greater than a predetermined threshold (“YES” at step740), method 700 proceeds to step 742. At step 742, a determination ismade as to whether image degradation has occurred, based on the resultof the determination at step 740. For example, image degradation mayinclude the images being distorted, out of focus, partially or whollyoccluded, and/or a mismatch between the images captured by left-eye lens210 and right-eye lens 220, thereby causing a stereoscopic visualdistortion even if the images, when viewed separately, do not appeardistorted. Thereafter, method 700 proceeds to step 750, where anotification is generated and provided to the user indicating that imagedegradation may have occurred. The notification may be displayed, forexample, via display device 44 or portable display device 45 of userinterface 40, and/or be provided audibly or tactilely via gimbals 70.

Following step 750, method 700 proceeds to step 755 where feedback isprovided to the user indicating how image quality may be improved. Forexample, the feedback provided may be in the form of a notification viadisplay device 44 or portable display device 45, that the user shouldremove and clean one or both of lenses 210, 220 in order to improveimage quality. After feedback is provided indicating how to improveimage quality at step 755, the method 700 proceeds to step 760 where itis determined whether the image quality can be improved during thesurgical procedure. For example, it may be determined whether imagecapture device 200 needs to be removed from surgical site “S” andcleaned, or be replaced. Those skilled in the art will envision variousother actions that may be taken to improve the image quality of imagescaptured by image capture device 200, and thus, for purpose of brevity,all such alternative actions will not be described here. If it isdetermined at step 760 that the image quality cannot be improved duringthe surgical procedure (“NO” at step 760), the method 700 proceeds tostep 775 where the surgical procedure ends. Alternatively, if it isdetermined at step 760 that the image quality can be improved during thesurgical procedure (“YES” at step 760), the method 700 proceeds to step765.

At step 765 it is determined whether the image quality has beenimproved. For example, the processes described above with reference tosteps 745, 747, 730, 735, and 740 may be repeated to determine whetherthe image quality of images captured by image capture device 200 hasbeen improved. If it is determined at step 765 that the image qualityhas not been improved (“NO” at step 765), processing returns to step755. Alternatively, if it is determined at step 765 that the imagequality has been improved (“YES” at step 765), the method 700 proceedsto step 770.

At step 770 it is determined whether the surgical procedure has beencompleted. For example, it may be determined whether the user hasprovided an instruction and/or indication that the surgical procedurehas been completed. If it is determined at step 770 that the surgicalprocedure has not been completed (“NO” at step 770), processing returnsto step 745. Alternatively, if it is determined at step 770 that thesurgical procedure has been completed (“YES” at step 770), processingends.

Referring back to the computer-readable media of FIG. 1 , memory 35includes any non-transitory computer-readable storage media for storingdata and/or software that is executable by processor 32 and whichcontrols the operation of controller 30. In an embodiment, memory 35 mayinclude one or more solid-state storage devices such as flash memorychips. Alternatively or in addition to the one or more solid-statestorage devices, memory 35 may include one or more mass storage devicesconnected to processor 32 through a mass storage controller (not shown)and a communications bus (not shown). Although the description ofcomputer-readable media contained herein refers to a solid-statestorage, it should be appreciated by those skilled in the art thatcomputer-readable storage media can be any available media that can beaccessed by processor 32. That is, computer readable storage mediaincludes non-transitory, volatile and non-volatile, removable andnon-removable media implemented in any method or technology for storageof information such as computer-readable instructions, data structures,program modules or other data. For example, computer-readable storagemedia includes RAM, ROM, EPROM, EEPROM, flash memory or other solidstate memory technology, CD-ROM, DVD, Blu-Ray or other optical storage,magnetic cassettes, magnetic tape, magnetic disk storage or othermagnetic storage devices, or any other medium which can be used to storethe desired information and which can be accessed by controller 30.Further aspects of the system and method are described in U.S. Pat. No.8,828,023 entitled “MEDICAL WORKSTATION,” filed on Nov. 3, 2011, theentire contents of all of which are hereby incorporated by reference.

Detailed embodiments of devices, systems incorporating such devices, andmethods using the same have been described herein. However, thesedetailed embodiments are merely examples of the disclosure, which may beembodied in various forms. Therefore, specific structural and functionaldetails disclosed herein are not to be interpreted as limiting, butmerely as a basis for the claims and as a representative basis forallowing one skilled in the art to employ the present disclosure invirtually any appropriately detailed structure.

1. (canceled)
 2. A method for detecting image degradation during asurgical procedure, the method comprising: receiving an image of asurgical instrument; obtaining a characteristic of a test patterndisposed on an outer surface of the surgical instrument, wherein thetest pattern includes a plurality of straight-edge portions defining twozones of different and contrasted luminance; processing the image toidentify the test pattern, which is captured in the received image andwhich is disposed on the outer surface of the surgical instrument;determining a characteristic of the identified test pattern in the imageof the surgical instrument based on a luminance of a first zone of thetwo zones of the test pattern and a second zone of the two zones of thetest pattern; calculating a difference between the determinedcharacteristic and the obtained characteristic; and generating an imagedegradation notification based on the difference.
 3. The methodaccording to claim 2, further comprising capturing the image of thesurgical instrument by an image capture device.
 4. The method accordingto claim 3, wherein the image capture device is a stereoscopic endoscopeincluding a left-eye lens and a right-eye lens.
 5. The method accordingto claim 2, further comprising obtaining the characteristic of the testpattern based on a baseline image of the test pattern.
 6. The methodaccording to claim 2, wherein the characteristic of the test pattern isobtained from a lookup table or a database.
 7. The method according toclaim 2, wherein the test pattern corresponds to a type of the surgicalinstrument.
 8. The method according to claim 2, further comprisingdetermining the characteristic of the identified test pattern based on amodulation transfer function (MTF) on the test pattern.
 9. The methodaccording to claim 8, wherein the test pattern includes black bars andwhite bars.
 10. The method according to claim 9, wherein the MTF is apercentage of contrast between the black bars and the white bars. 11.The method according to claim 2, further comprising periodicallyreceiving another image of the surgical instrument at a predeterminedinterval.
 12. A system for detecting image degradation during a surgicalprocedure, the system comprising: a surgical instrument; an imagecapture device configured to capture an image of the surgicalinstrument, the image of the surgical instrument including a testpattern disposed on an outer surface of the surgical instrument; atleast one processor coupled to the image capture device; and a memorycoupled to the at least one processor and having instructions storedthereon, that, when executed by the at least one processor, cause the atleast one processor to: obtain a characteristic of the test pattern;process the image to identify the test pattern, which is captured in theimage and which is disposed on the outer surface of the surgicalinstrument; determine a characteristic of the identified test pattern inthe image of the surgical instrument; calculate a difference between thedetermined characteristic and the obtained characteristic; and generatean image degradation notification based on the difference.
 13. Thesystem according to claim 12, wherein the image capture device is astereoscopic endoscope including a left-eye lens and a right-eye lens.14. The system according to claim 12, wherein the characteristic of thetest pattern is obtained based on a baseline image of the test pattern.15. The system according to claim 12, wherein the characteristic of thetest pattern is obtained from a lookup table or a database.
 16. Thesystem according to claim 12, wherein the test pattern corresponds to atype of the surgical instrument.
 17. The system according to claim 12,wherein the characteristic of the identified test pattern is determinedbased on a modulation transfer function (MTF) on the test pattern. 18.The system according to claim 17, wherein the test pattern includesblack bars and white bars.
 19. The system according to claim 18, whereinthe MTF is a percentage of contrast between the black bars and the whitebars.
 20. The system according to claim 12, further comprising whereinthe at least one processor periodically receiving another image of thesurgical instrument at a predetermined interval.
 21. A non-transitorycomputer-readable medium storing instructions that, when executed by aprocessor, cause the processor to: receive an image of a surgicalinstrument; obtain a characteristic of a test pattern disposed on anouter surface of the surgical instrument; process the image to identifythe test pattern, which is captured in the received image and which isdisposed on the outer surface of the surgical instrument; determine acharacteristic of the identified test pattern in the image of thesurgical instrument; calculate a difference between the determinedcharacteristic and the obtained characteristic; obtain baseline imagesof one or more straight outer edges of the surgical instrument; comparea characteristic of one or more straight outer edges of the surgicalinstrument in the images to a characteristic of the one or more straightouter edges of the surgical instrument in the baseline images, theimages of the surgical instrument being received subsequent to theobtaining of the baseline images and being received while the surgicalinstrument is disposed at a surgical site in a patient; determinewhether the images of the surgical instrument are degraded, based on thecomparison of the characteristic of one or more straight outer edges ofthe surgical instrument in the images and the characteristic of the oneor more straight outer edges of the surgical instrument in the baselineimages; and generate an image degradation notification based on thedifference between the determined characteristic and the obtainedcharacteristic, and in response to a determination that the images ofthe surgical instrument are degraded.